The present invention relates generally to the field of automated testing of mobile software applications, and more particularly to analysis of reports generated by automated tests.
Automated testing of software applications is a common tool for verifying the functionality of new applications by simulating interactions with a user by executing one or more test steps and monitoring the result of each test step. Different forms of automated testing have different limitations and benefits depending on the goal(s) of the particular test. Two common forms of automated testing are performance testing, such as mobile resource usage monitoring and profiling testing or response time, in which a computer executes a series of test steps while recording information on one or more physical computing resources, and functional testing, in which a series of test steps are executed to determine whether the application under test responded to the test step in an appropriate manner.
In performance testing, a computer executes a series of test steps in an environment that simulates the type of device on which the application can be run. During the execution, an automated testing tool monitors various hardware components, such as CPU usage, battery metrics, RAM usage, virtual memory usage, network traffic, etc. During the execution of the performance test, the testing tool periodically takes snapshots in order to generate a visual record of what the application displays to a user at a given point in time. In a performance test, the testing tool also monitors events, such as when the application under test attempts to access a third-party service (e.g., a remote server, or a network), and records the timing of the event when the testing tool detects one.
In functional testing, software developers/testers supplement the code of the application under test with additional instructions, or instrumentation, to facilitate the functional test. The process of adding the additional code is known as “instrumenting” or “tainting” the application under test. The instrumentation consists of instructions to, for example, determine whether a particular test step passed (i.e., performed as expected), determine which particular method(s) were being executed in the test step, capture a snapshot of an application under test graphical user interface, and/or determine when particular events, such as accessing a remote server, take place.